Mapping and Validating Case Specific Cognitive Models

This study proposes a visual methodology to validate complex solution processes in the context of ill-structured problem solving. This experiment is anchored in the context of medical case-based teaching. The case validation activity proposed is modeled on the authentic case presentation practice performed by physicians. We are using a computer-based learning environment (BioWorld) to present a standardized set of cases to expert teachers who are asked to solve the case and do a think-aloud protocol while solving the cases. We are developing a methodology that addresses both knowledge elicitation as well as knowledge validation for solving and reflecting on ill-structured medical problems. More precisely, this study examines the effectiveness of visual support tools to help physicians verify their diagnostic thinking. In so doing our goal is to build and validate case specific cognitive models. Introduction Problem based learning (PBL) is not a new approach in medical education, it is used to teach clinical reasoning and problem solving skills in a number of medical schools (Barrow, 1994; Barrows & Tamblyn, 1980; Koschman, Kelson, Feltovich, & Barrow, 1996). The use of cases for teaching is as old as storytelling and Cox (2001) argues that this way of transmitting knowledge provides a meaningful framework to embed all the objectives and sub-objectives related to a complex patient case. A case presentation in medicine generally consists of a detailed analysis of a patient case but depending on the instructor’s prior experience and the facilities in which the patient is seen the solution to these cases varies substantially. Case development work for BioWorld (a computerbased learning environment (Lajoie, Lavigne, Guerrera, & Munsie, 2001)) led us to note significant differences in the thinking and decision making processes involved in complex case solution. Data on the case creation phase demonstrated both validity and reliability issues when working with medical staff and students. This lack of consistency forced us to address the issue of validity and reliability of ill-structured solutions in a more systematic manner. Case presentation activities are used to teach diagnostic reasoning. Diagnostic reasoning about patient cases share the same components of ill-structured problem solving as defined by Jonassen (1997) in that solving patient cases involve a) plenty of unknown elements, b) there is not one correct unambiguous solution, c) there is more than one way to reach a diagnosis and there are usually multiple ways to reach an acceptable answer (often referred as differential diagnosis) c) there is no absolute criteria or way to validate the answer, and d) case resolution often involves ethical and personal judgments. This research aims at modeling and building on the case presentation activity that occurs in medical education. We do not intend to replace or compete with the face-to-face case presentations but it aims at documenting and building on key elements related to this practice. In this paper we first describe the instructional context and the computer-based learning environments we use to support and study diagnostic reasoning. We then explain how the validation activity became a key element of the case creation process. We also explain why and how the sampling of detailed solution processes and explanations of expert teachers can lead to the construction of cognitive models. Cognitive tools to support and study diagnostic reasoning BioWorld, is a computer-based learning environment that was first designed to promote scientific reasoning in high school students. It provides a realistic environment for students to learn about diseases through solving specific patient cases (Lajoie et al., 2001). Solving a patient case in BioWorld not only consists of submitting a good diagnostic but it also requires students to select and organize evidence that supports and justifies decisions through the case resolution process. Pilot work with medical students, residents and staff physicians was conducted using BioWorld cases and conclusions recommended the use of this learning environment for medical education (Faremo, 2004). One key aspect of adapting BioWorld to a medical audience is to revise and construct cases at an appropriate level of difficulty. In our attempts to create and develop valid cases in medical education, we have experimented with different methodologies and scenarios to structure case creation. The companion authoring tool, CaseBuilder (Lajoie et al., 2001) which was designed to allow both instructors and researchers to modify cases easily also enables us to explore instructional activities for content creation and revision. Creating cases for an interactive computer environment implies documenting not only the questions and information related to the acceptable answer but it also requires the inclusion of plausible distracters or possible questions learners might have while trying to solve the case. Creating a case can also be referred to as a problem generation activity, which is an instructional technique that requires the learner to assemble or construct a problem. In a problem generating task the learner needs to choose a specific case to construct or modify the problem they choose to explore and analyze if all the elements are defensible and could make sense for potential problem solvers. Silver (1994) includes both problem modification and problem construction in his working definition of the technique. The problem creator has to make sense of a situation and determine which elements can contribute to the solution and which other elements need to be present as distracters to increase the level of difficulty of the problem. In this context the case builder becomes a cognitive tool that supports our exploration of this learner-centered knowledge building activity. Our data on the case creation phase demonstrated both validity and reliability issues when working with medical staff and students. As we raised the level of complexity of the cases in BioWorld we encountered challenges in the design and validation of solutions for these complex cases. As mentioned above, the solution to a case in BioWorld does not only consist of the final answer but it also requires a list of prioritized supporting evidences related to this answer. Consequently in the case creation activity, when medical students were constructing cases they had to provide and list the evidence supporting a good diagnosis. We found discrepancies between their hypothetical answer (answers they had planned) and their actual solution (the one recorded when they ran through the case in BioWorld). We first hypothesized that students were maybe not qualified enough to provide a clear answers so we asked a medical expert to do the same case twice. The expert was not aware of having to solve the same case twice; patient names were changed, a 10 day delay between testing occurred and we presented the expert with other similar cases in between the two cases in question. Again we obtained non-identical answer for a relatively simple case of diabetes. Our last attempt to address the inconsistent supporting evidence was to ask a second medial expert to do the exact same diabetes’ case. Results were consistent with our two previous experiences and this lack of consistency forced us to address the issue of validity and reliability of solutions in a more systematic manner. Tracking expert solution processes and explanations To address the variability of the case’s solutions we decided to construct a case validation activity. The simple list of evidence for justifying and explaining the answer was not sufficient to show where and how expert differed in their problem solving processes. Therefore the validation activity was designed to scrutinize every step along the way by including think-aloud protocols of individual participant. We justify this investment of time and resources by using expert teachers that have the ability and experience to solve and explain theirs though processes to others. Additionally, the validation activity has revealed itself to be a key component of the case creation process. The activity provides motivation and feedback to the medical student who acts as case creator. On the other hand when teachers solve cases created by students it gives them clear examples of students’ misunderstanding of content and interrelationships of the different components involved in the diagnostic of cases. The validation activity consists of a simulation of a case presentation for medical teachers. Participants are asked to think aloud (do a think-aloud protocol) and provide explanations as they solve a case in BioWorld. The level of the cases and their explanation is at the undergraduate level. From a research perspective, we want to capture and record the strategies experts use to synthesize the information about a disease as well as how they structure and communicate this information in both oral and written forms. Expert teacher can provide us with relatively clear “path” of the decision process as well as explanations and verbalization about the metacognitive strategies they use while solving the cases. This validation activity will be used as a blueprint to build a cognitive model for each of our cases. Sampling individual and collective problem representation Protocol analyses are used to explore domain knowledge, to describe what are key elements and how knowledge is structured and used during a problem-solving task (Ericsson & Simon, 1993). Whereas protocol analyses of well-defined problems can result in clear problem solving sequences the analysis of ill-structured problems can be more complex given there is more then one way to reach a solution. We do not aim at conducting an exhaustive task analysis of all the possible solutions path or options but to sample and represent two to five solution paths for each case. The goal is not simply to build an expert path and use it to compare to novices’ performance but to build a partial problem space representation that can evolve as more people do theses cases. The visual representations are built to offer a short summary of the though processes with the relative importance of specific steps to the resolution of the case. When interacting with theses representations participants can “zoom in” and open sub-layers to access details, related explanation and exact verbal transcript from the verbal protocol. The problem space is constructed in three phases. The initial representation built by the researcher summarizes the decision-making process. It is used with experts to have them validate and reflect on the resolution path of the problem. We ask expert to first validate the summary of their case resolution and then select and categorize section of their decision path. Experts are asked to select which elements are absolutely necessary to the case resolution, which ones are necessary and which one adds useful information. The second version incorporates the changes and categorization done by the experts. This categorization of decision path reveals the relative weight of specific steps and facilitates a comparison between experts. In the third version of the representation we merge experts path to show similarities and differences in the sequence of decisions leading to acceptable answer(s) for a specific case. Goal and purposes of the visual representation This validation activity serves multiple purposes. The visual representation is a tool to build and to communicate partial data to participants. As our initial participants are expert medical teachers we hope to extract pedagogical models for teaching specific cases and not only experts’ case solution processes. Their experience teaching concepts related to each cases and their ability to predict what learners at the undergraduate level can understand will help us validate and improve content. The actual visual representation could also be incorporated into the computer-based learning environment to teach students. However we hope to use these qualitative blueprints to implement better scaffolding and feedback mechanism into our computer-based learning environment. Situating the methodology The use of a diagram, for the partial analysis of data is not a common procedure but Henderson, Yerushalmi, Heller & Kuo (2003) have found that visual maps are useful to analyze complex interview data. They found that concept maps reflected participants’ conceptual understanding of the topic, clearly showed relationships between concepts and were useful to show similarities and differences between participants. The use of the term ‘concept map’ by these authors can be misleading as they do not use it in the way Novak and Cañas (2006) describe in their work. Henderson, Yerushalmi, Heller & Kuo used the Cmap software as a knowledge visualization tool to provide a visual overview of their protocol analysis. The diagram is not constructed by the participant but by the researcher from the verbal protocol. We chose Henderson et al.’s technique as a starting point to develop our own methodology to show a clear link between raw data and the participants’ conceptual and procedural knowledge while solving a case.